Papers with evaluation strategies
MUG: Interactive Multimodal Grounding on User Interfaces (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior studies modeled multimodal UI grounding in one round, but such an interaction is inherently iterative. |
| Approach: | They propose a task where a user and an agent collaborate on an interface screen . they use a dataset of 77,820 sequences of human user-agent interaction on mobile interfaces . |
| Outcome: | The proposed task improves the absolute task completion by 18% over the entire test set and 31% over the challenging split. |
Returning the N to NLP: Towards Contextually Personalized Classification Models (2020.acl-main)
Copied to clipboard
| Challenge: | a recent study shows that NLP models treat language as universal, but that it is based on sociolinguistic research. |
| Approach: | They propose to incorporate user-dependent, contextual personal and social aspects into neural NLP models by means of socially contextual personalization. |
| Outcome: | The proposed approach could be adapted to better personalize the language of users . it outlines a possible direction to incorporate these aspects into neural NLP models . |
Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Audio-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research focused on model-specific adversarial methods, but real-world applications demand a more generalizable approach to audio adversarials. |
| Approach: | They propose a Chat-Audio Attacks benchmark to evaluate LALMs' robustness . they propose standard evaluation, GPT-4o-based evaluation and human evaluation . |
| Outcome: | The proposed benchmark aims to explore the robustness of six state-of-the-art LALMs with voice interaction capabilities. |
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering tasks are one of the most commonly used tasks for evaluating Large Language Models (LLMs). |
| Approach: | They analyze whether existing answer extraction methods are aligned with human judgment and how they are influenced by answer constraints in the prompt across different domains. |
| Outcome: | The proposed evaluation strategies can be inconsistent with human judgment, and can lead to inaccurate and misleading comparisons. |